Repository navigation
Log: a commit covered by the fsync in flight is not refused when the log fails - #2
Merged
Merged
Conversation
…log fails The append of one commit failed with a later commit already behind it, so the log is failed for good. A commit parked in sync_staged whose record the running fsync was making durable saw `failed` first and returned IOERR_FSYNC; the leader then finished, published the epoch and the commit was visible (and in the log after a reopen) although it was told that it had failed. mw_rollbackrace showed it as "t4 has 80 rows, 79 acknowledged" (2 in 150 runs on two CPUs, also on main; every run on the CI runner of the PR). A waiter that finds the log failed now waits while an fsync is in flight and decides after it: durable -> acknowledged, otherwise refused. No fsync begins once the log is failed, so what the running one does not cover stays invisible. mw_holesync reproduces it in every run (x covered by the held fsync, f fails, s behind the hole): without the change w is refused and visible. The fsync is held by building multiwriter_log.c with -Dfsync=inject_fsync, as the other tests do with pwritev and malloc. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Il problema
mw_rollbackracefalliva in CI (linux-x86_64 build + test, runpull_requestdi #1, 3 tentativi su 3) con:Il fallimento è preesistente: l'ho riprodotto su
main(e8acd7e) in un container Linux con 2 CPU pinnate (2 su 150 esecuzioni).Causa
In
sync_staged:fsyncgià in volo sta per renderlo durevole.failedper primo e restituisceIOERR_FSYNC; poi l'fsynctermina, il leader pubblica l'epoch e la riga di W diventa visibile (e presente nel log dopo una riapertura) pur essendo stata dichiarata fallita.Correzione
Chi trova il log failed aspetta finché c'è un
fsyncin volo e decide dopo: durevole → confermato, altrimenti rifiutato. Dopo il fallimento non parte nessun altrofsync(tutti i waiter escono), quindi ciò che l'fsyncin volo non copre resta invisibile.8 righe in
src/multiwriter_log.c.Verifica
mw_rollbackrace, Linux aarch64, 2 CPU pinnatemain), 3 su 142 (strumentato)mw_holesync(nuovo, deterministico)w: commit=10, visible=1in ogni runmake test,make test-mpmw_holesync,mw_rollbackracemw_holesyncmw_holesyncforza la sequenza con unfsyncbloccato (multiwriter_log.ccompilato con-Dfsync=inject_fsync, comemw_stagednormalfa conpwritev): x e w sono coperti dall'fsynctenuto, f fallisce, s è dietro il buco. Senza la correzione w è rifiutato ma visibile.Da sapere
review-fixes: il branch contiene anchec663295ec1cf57edella Fix rebase conflicts, log and ATTACH durability, recovery and row diff OOM #1; il diff di questa PR è solo57351a1.MW_FAULT_DELAY_US, 1 s) per ordinare gli eventi, conalarm(30): su un runner molto lento un margine potrebbe stringersi e il sintomo sarebbe un hang visibile.🤖 Generated with Claude Code